Skip to content

OCPBUGS-113581: Raise status polling timeout and write bound - #31553

Open
redhat-chai-bot wants to merge 1 commit into
openshift:release-4.20from
redhat-chai-bot:cherry-pick-31534-to-release-4.20
Open

OCPBUGS-113581: Raise status polling timeout and write bound#31553
redhat-chai-bot wants to merge 1 commit into
openshift:release-4.20from
redhat-chai-bot:cherry-pick-31534-to-release-4.20

Conversation

@redhat-chai-bot

@redhat-chai-bot redhat-chai-bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Cherry-pick of #31534 to release-4.20.

Fixes flaky test [sig-network][Feature:Router] The HAProxy router converges when multiple routers are writing conflicting status by raising two thresholds in test/extended/router/stress.go:

  1. Polling timeout: 2 min → 5 min — the leader election lease lasts 1 minute; under heavy contention a replica may need 3+ lease cycles to update all routes.
  2. Write-count bound: 50 → 75 — the async contention detector can lag behind update events, allowing extra writes before suppression kicks in.

This test caused 367 failures in a single 7-day window before the fix landed on main. The 4.20 nightly e2e-rosa-sts-ovn job continues to hit this flake (most recently with 52 writes vs the 50 limit).

References

AI-generated. Review for accuracy.

@dustman9000 requested in Slack thread

@jcmoraisjr @redhat-chai-bot
Raise status polling timeout and write bound
279f577 
The 3-replica conflicting router stress test was intermittently failing
in two independent places.

1. Polling timeout 2m -> 5m: the leader election lease lasts 1 minute.
   Under normal conditions all route statuses are written within 1-2
   lease cycles, which fit in the previous 2-minute timeout. Under
   heavier contention a router replica may need to re-acquire the lease
   3 or more times until all routes are updated.

2. Write upper bound 50 -> 75: the per-route contention detector runs
   asynchronously and can lag behind update events, allowing a few extra
   writes before suppressing further updates. The previous limit of 50
   caused intermittent failures; 75 is a more conservative ceiling.

https://redhat.atlassian.net/browse/OCPBUGS-99536
@openshift-ci-robot openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 24, 2026
@openshift-ci-robot

openshift-ci-robot commented Aug 24, 2026

Copy link
Copy Markdown

@redhat-chai-bot: This pull request references Jira Issue OCPBUGS-99536, which is invalid:

  • expected the bug to target either version "4.20.z." or "openshift-4.20.z.", but it targets "5.1.0" instead
  • expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is ON_QA instead
  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected Jira Issue OCPBUGS-99536 to depend on a bug targeting a version in 4.21.0, 4.21.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Cherry-pick of #31534 to release-4.20.

Fixes flaky test [sig-network][Feature:Router] The HAProxy router converges when multiple routers are writing conflicting status by raising two thresholds in test/extended/router/stress.go:

  1. Polling timeout: 2 min → 5 min — the leader election lease lasts 1 minute; under heavy contention a replica may need 3+ lease cycles to update all routes.
  2. Write-count bound: 50 → 75 — the async contention detector can lag behind update events, allowing extra writes before suppression kicks in.

This test caused 367 failures in a single 7-day window before the fix landed on main. The 4.20 nightly e2e-rosa-sts-ovn job continues to hit this flake (most recently with 52 writes vs the 50 limit).

References

AI-generated. Review for accuracy.

@dustman9000 requested in Slack thread

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Aug 24, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: e9444587-10ff-4b7e-870b-a3ce975d97e3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Aug 24, 2026
@openshift-ci
openshift-ci Bot requested review from candita and gcs278 August 24, 2026 15:35
@openshift-ci

openshift-ci Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: redhat-chai-bot
Once this PR has been reviewed and has the lgtm label, please assign grzpiotrowski for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@redhat-chai-bot redhat-chai-bot changed the title OCPBUGS-99536: Raise status polling timeout and write bound OCPBUGS-113581: Raise status polling timeout and write bound Aug 24, 2026
@openshift-ci-robot openshift-ci-robot removed the jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. label Aug 24, 2026
@openshift-ci-robot

openshift-ci-robot commented Aug 24, 2026

Copy link
Copy Markdown

@redhat-chai-bot: This pull request references Jira Issue OCPBUGS-113581, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected Jira Issue OCPBUGS-113581 to depend on a bug targeting a version in 4.21.0, 4.21.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Cherry-pick of #31534 to release-4.20.

Fixes flaky test [sig-network][Feature:Router] The HAProxy router converges when multiple routers are writing conflicting status by raising two thresholds in test/extended/router/stress.go:

  1. Polling timeout: 2 min → 5 min — the leader election lease lasts 1 minute; under heavy contention a replica may need 3+ lease cycles to update all routes.
  2. Write-count bound: 50 → 75 — the async contention detector can lag behind update events, allowing extra writes before suppression kicks in.

This test caused 367 failures in a single 7-day window before the fix landed on main. The 4.20 nightly e2e-rosa-sts-ovn job continues to hit this flake (most recently with 52 writes vs the 50 limit).

References

AI-generated. Review for accuracy.

@dustman9000 requested in Slack thread

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@dustman9000

dustman9000 commented Aug 24, 2026

Copy link
Copy Markdown
Member

/jira refresh

@openshift-ci-robot

openshift-ci-robot commented Aug 24, 2026

Copy link
Copy Markdown

@dustman9000: This pull request references Jira Issue OCPBUGS-113581, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected Jira Issue OCPBUGS-113581 to depend on a bug targeting a version in 4.21.0, 4.21.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-robot

openshift-merge-robot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Fix included in release 5.1.0-0.nightly-2026-08-24-111752

@redhat-chai-bot

redhat-chai-bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

/retest-required

AI-generated. Review for accuracy.

@openshift-ci

openshift-ci Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@candita candita Awaiting requested review from candita
@gcs278 gcs278 Awaiting requested review from gcs278

Assignees

No one assigned

Labels

jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@redhat-chai-bot @openshift-ci-robot @dustman9000 @openshift-merge-robot @gcs278 @jcmoraisjr